Method and apparatus for correcting white balance, method for correcting density and recording medium on which program for carrying out the methods is recorded

ABSTRACT

The white balance correction method and apparatus estimate a color temperature of a photographing light source with which a color image has been taken by using at least gray and/or skin color information contained in an input color image and correct image signals of the color image based on the estimated color temperature. The density correction method multiplies image signals of respective pixels in an input color image by set coefficients to detect pixels having the multiplied image signals in the vicinity of a blackbody locus curve of skin color as skin color candidate pixels and assigns an average obtained for predetermined color signals from the skin color candidate pixels detected to a predetermined density of a color corresponding to the color signals on a print. The recording medium on which one or both of the white balance correction method and the density correction method are recorded in a computer-readable manner as a program to be executed by a computer.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to a technique for correcting whitebalance and density of a color image when digital image processing isperformed on an input image signal to form a photographic print, moreparticularly, to a method and an apparatus for correcting white balance,a method for correcting density and a recording medium on which aprogram for carrying out these methods is recorded.

[0003] 2. Description of the Related Art

[0004] The conventional silver salt photography generally uses an analogexposure (area exposure or direct exposure) system to obtain a print.That is, a developed negative film is positioned at a predeterminedprinting position and irradiated with light from a white light source(halogen lamp or the like), and light transmitted through the negativefilm is imaged on photographic paper, thus making an exposure.

[0005] In recent years, printing apparatuses using digital exposure,i.e., digital photoprinters, have been put to practical use. Digitalphotoprinters perform a process in which an image recorded on aphotographic film such as a negative film or a color reversal film isphotoelectrically read to obtain a digital signal representing the readimage, various kinds of image processing is performed on the digitalsignal to obtain image data for the recording purposes, scanningexposure using recording light modulated according to this image data isperformed on a photosensitive material to form an image (latent image)on the photosensitive material to obtain a (finished) print.

[0006] Such digital photoprinters process images as digital image dataand can therefore perform image processing not only on photographedimages on photographic films but also on images photographed withdigital still cameras (DSC) or the like, image data recorded in the formof digital data on various kinds of recording media, e.g., magneticrecording media, such as CD-R, flexible disks, and removable hard disks(Zip, Jaz, etc.), and MO disks (magneto-optical recording media) tooutput prints.

[0007] Color negative films have been commonly used by typical users.The conditions of photography of original images on negative films arenot always uniform. For example, subjects are photographed in light fromvarious light sources, e.g., daylight and fluorescent lamp. Therefore,in case of forming prints from a developed negative film when images inthe developed negative film is printed without changing thecharacteristics of light from a printing light source, color tints oflight from photographing light sources in which subjects have beenphotographed are directly reflected in the prints, which leads toforming undesirable prints in some cases.

[0008] Various systems have therefore been contrived to adjust whitebalance on the prints. A typical example of such systems is a large areatransmission density (LATD) method based on the Evans' theorem(hypothesis) stating that the average of all colors in the world isgray. The LATD is the average transmission density through an entireframe on a photographic film. In the LATD method, the LATD of each frameon a color negative film is measured and the characteristics of lightfrom a printing light source are changed according to the red, green,and blue density levels so that the average color on the print is madeclose to gray.

[0009] On the other hand, digital still cameras (DSCS) have recentlycome into widespread use. An image forming process in a DSC and an imageforming process using a color negative film can be regarded as the samewith respect to photographing a scene but differ essentially in that aDSC image itself is an object of appreciation while an image on a colornegative film is not directly viewed as an object of appreciation. DSCimages themselves, therefore, must be fine, well-white-balanced imageswhen viewed before being output as prints. DSCs are also used forphotography with various light sources like cameras using color negativefilms and are unable to reliably obtain a satisfactory image without afunction for correcting white balance. Therefore almost all the recentDSCs have an auto white balance (AWB) function for automaticallycorrecting white balance.

[0010] The above-described LATD method has been practiced with somemeasure of success but it is responsible for production of undesirableprints. One of the printing failures due to the LATD method is colorfailure that imbalance of a color occurs in a print. For example, in acase where white balance correction based on the LATD method is made onan image obtained by photographing a woman in red dress, a cyan color,which is a complementary color to red, is added throughout the entireframe to make the entire frame close to gray, thereby reducing thevividness of the red of the dress and making the woman's face pale.

[0011] In a case where the entire frame area of a photographed scene isreddish, it is impossible to ascertain whether the reddishness isascribable to the light source or the subject. In this case, the LATDmethod achieves success in correction if the light source is the cause,but it causes such a color failure described above if the subject is thecause.

[0012] The above-mentioned AWB function of DSCs is essentially based onthe Evans' theorem like the negative film/printing system and entailsthe same problem as that of white balance correction based on the LATDin the negative film/printing system.

[0013] That is, while about 60 to 70% of DSC images after AWB areobtained as well-color-balanced good images as average performance ofthe Evans' theorem, the remaining 30 to 40% of the DSC images need someadditional color balance correction because of AWB function failure. Ifwhite balancing is not performed at the time of printing from the DSCimages, about 30 to 40% of resulting prints are unsatisfactory,unacceptable prints.

[0014] As described above, the conventional art LATD method based on theEvans' theorem makes white balance correction by assuming that theaverage of colors through the entire frame of an image is gray, but itis not sufficiently effective in correcting white balance because of itsinability to find true gray in an image, and often produces a contrarycorrection effect.

[0015] Further, when printing, if LATD method as above is employed toperform density correction through the entire image, the density of aprincipal subject in the image will be influenced by sceneconfigurations and a print may be formed with an improper density. Inorder to overcome such a problem, it has been proposed to detect aprincipal subject (a human face in many cases) in an image and determinethe print density in accordance with the density of the detectedsubject. The judgment on whether or not the print density is proper ismade based on the density of a principal subject rather than the densitythrough an entire frame. Consequently, it is important to detect a humanface as a principal subject and correct the density of the entire imageso that the density of the face may be proper.

[0016] Detection of a human face as a principal subject is generallyperformed employing shape recognition. However, it is very difficult atpresent to detect human faces with high accuracy by employing suchmeasures. It has also been attempted to detect human faces using colorinformation, where difficulties may again occur, if the type of a lightsource is unknown.

SUMMARY OF THE INVENTION

[0017] The present invention has been devised in view of theabove-mentioned problem of the prior art and, it is a first object ofthe present invention to provide a method and an apparatus for whitebalance correction, which is a technique for realizing white balancecorrection appropriately and with high yield in applying digital imageprocessing to inputted image data to form prints, and a recording mediumhaving recorded therein a program for executing this program.

[0018] In addition, the present invention has been devised in view ofthe above-mentioned problem of the prior art and, it is a second objectof the present invention to provide a method and an apparatus fordensity correction, which is a technique for detecting an area of skincolor out of an image to obtain an appropriate print density based oninformation on the area of skin color if a light source is unknown, anda recording medium having recorded therein a program for executing thismethod.

[0019] In order to attain the first object, the first aspect of thepresent invention provides a white balance correction method, comprisingthe steps of estimating, by using at least gray and/or skin colorinformation contained in an input color image, a color temperature of aphotographing light source with which the color image has been taken,and correcting image signals of the color image based on the estimatedcolor temperature.

[0020] Preferably, only the gray and skin color information is used whencorrecting the image signals of the color image.

[0021] Preferably, the estimating step includes multiplying the imagesignals of respective pixels in the input color image by setcoefficients, setting pixels having the multiplied image signals in thevicinity of a blackbody locus curve of skin color as skin colorcandidate pixels and/or pixels having the multiplied image signals inthe vicinity of a blackbody locus curve of gray as gray candidatepixels, optimizing the set coefficients so that the number of the skincolor candidate pixels and/or the gray candidate pixels is maximized,obtaining a group of skin color candidate pixels and/or a group of graycandidate pixels by using the optimized set coefficients, and estimatingthe color temperature of the photographing light source from an averagecolor temperature of the group of skin color candidate pixels and/or anaverage color temperature of the group of gray candidate pixels, andwherein the image signals of the color image multiplied by the optimizedset coefficients are corrected by an amount corresponding to adifference between the estimated color temperature and a colortemperature of reference white.

[0022] Preferably, the color temperature of the photographing lightsource is estimated from the average color temperature of the group ofskin color candidate pixels and the average color temperature of thegroup of gray candidate pixels obtained by optimizing the setcoefficients so that the number of the skin color candidate pixels andthe gray candidate pixels is maximized.

[0023] Preferably, the estimating step includes multiplying the imagesignals of respective pixels in the input color image by setcoefficients, setting pixels having the multiplied image signals in thevicinity of a blackbody locus curve of skin color as skin colorcandidate pixels and pixels having the multiplied image signals in thevicinity of a blackbody locus curve of gray as gray candidate pixels,optimizing the set coefficients so that a difference between an averagecolor temperature of the skin color candidate pixels and an averagecolor temperature of the gray candidate pixels is minimized, obtaining agroup of skin color candidate pixels and a group of gray candidatepixels by using the optimized set coefficients, and estimating the colortemperature of the photographing light source from the average colortemperature of the group of skin color candidate pixels and the averagecolor temperature of the group of gray candidate pixels, and wherein theimage signals of the color image multiplied by the optimized setcoefficients are corrected by an amount corresponding to a differencebetween the estimated color temperature and a color temperature ofreference white.

[0024] Preferably, the estimating step includes multiplying the imagesignals of respective pixels in the input color image by setcoefficients, setting pixels having the multiplied image signals in thevicinity of a blackbody locus curve of skin color as skin colorcandidate pixels and pixels having the multiplied image signals in thevicinity of a blackbody locus curve of gray as gray candidate pixels,optimizing the set coefficients so that the number of the skin colorcandidate pixels and/or the gray candidate pixels is maximized and adifference between an average color temperature of the skin colorcandidate pixels and an average color temperature of the gray candidatepixels is minimized, obtaining a group of skin color candidate pixelsand a group of gray candidate pixels by using the optimizedcoefficients, and estimating the color temperature of the photographinglight source from the average color temperatures of the group of skincolor candidate pixels and the group of gray candidate pixels, andwherein the image signals of the color image multiplied by the optimizedset coefficients are corrected by an amount corresponding to adifference between the estimated color temperature and a colortemperature of reference white.

[0025] Preferably, the estimating step includes multiplying the imagesignals of respective pixels in the input color image by setcoefficients, setting pixels having the multiplied image signals in thevicinity of blackbody locus curve of skin color as skin color candidatepixels and pixels having the multiplied image signals in the vicinity ofa blackbody locus curve of gray as gray candidate pixels, optimizing theset coefficients so that the number of the skin color candidate pixelsand/or gray candidate pixels is maximized and a difference between anaverage color temperature of the group of skin color candidate pixelsand the group of gray candidate pixels is minimized, obtaining a groupof skin color candidate pixels and a group of gray candidate pixels byusing the optimized coefficients, dividing the group of skin colorcandidate pixels into a plurality of subgroups of skin color candidatepixels and dividing the group of gray candidate pixels into a pluralityof subgroups of gray candidate pixels, and estimating a colortemperature of the photographing light source from an average colortemperature of a subgroup of skin color candidate pixels with a highaverage color temperature among the plurality of subgroups of skin colorcandidate pixels and an average color temperature of a subgroup of graycandidate pixels with a high average color temperature among theplurality of subgroups of gray candidate pixels, and wherein the colorimage signals multiplied by the optimized coefficients are corrected bya difference between the estimated color temperature and a colortemperature of reference white.

[0026] Preferably, the estimating step includes multiplying the imagesignals of respective pixels in the input color image by setcoefficients, setting pixels having the multiplied image signals in thevicinity of blackbody locus curve of skin color as skin color candidatepixels and pixels having the multiplied image signals in the vicinity ofa blackbody locus curve of gray as gray candidate pixels, optimizingfirstly the set coefficients so that the number of the skin colorcandidate pixels and the gray candidate pixels is maximized and adifference between an average color temperature of the group of skincolor candidate pixels and an average color temperature of the group ofgray candidate pixels is minimized, estimating a first color temperatureof the photographing light source from the average color temperature ofa group of skin color candidate pixels and the average color temperatureof a group of gray candidate pixels obtained by the first optimization,optimizing secondly the set coefficients so that the number of the graycandidate pixels is maximized and a difference between an average colortemperature of the group of skin color candidate pixels and an averagecolor temperature of the group of gray candidate pixels is minimized,and estimating a second color temperature of the photographing lightsource from the average color temperature of a group of skin colorcandidate pixels and the average color temperature of a group of graycandidate pixels obtained by the second optimization, and wherein thecolor image signals multiplied by the optimized coefficients arecorrected by using both a first white balance correction signal and asecond white balance correction signal, the first white balancecorrection signal being adapted for correcting the color image signalsby a difference between the first estimated color temperature and acolor temperature of reference white, and the second white balancecorrection signal being adapted for correcting them by a differencebetween the second estimated color temperature and a color temperatureof reference white.

[0027] Preferably, the estimating step includes multiplying the imagesignals of respective pixels in the input color image by setcoefficients, setting pixels having the multiplied image signals in thevicinity of blackbody locus curve of skin color as skin color candidatepixels and pixels having the multiplied image signals in the vicinity ofa blackbody locus curve of gray as gray candidate pixels, optimizingfirstly the set coefficients so that the number of the gray candidatepixels is maximized and a difference between an average colortemperature of the group of skin color candidate pixels and an averagecolor temperature of the group of gray candidate pixels is minimized,estimating a first color temperature of the photographing light sourcefrom the average color temperature of a group of skin color candidatepixels and the average color temperature of a group of gray candidatepixels obtained by the first optimization, optimizing secondly the setcoefficients so that the number of the skin color candidate pixels ismaximized and a difference between an average color temperature of thegroup of skin color candidate pixels and an average color temperature ofthe group of gray candidate pixels is minimized, estimating a secondcolor temperature of the photographing light source from the averagecolor temperature of a group of skin color candidate pixels and theaverage color temperature of a group of gray candidate pixels obtainedby the second optimization, and wherein the color image signalsmultiplied by the optimized coefficients are corrected by using both afirst white balance correction signal and a second white balancecorrection signal, the first white balance correction signal beingadapted for correcting the color image signals by a difference betweenthe first estimated color temperature and a color temperature ofreference white, and the second white balance correction signal beingadapted for correcting them by a difference between the second estimatedcolor temperature and a color temperature of reference white.

[0028] Preferably, the image signals of respective pixels in the inputcolor image are multiplied by set coefficients and, as a result of themultiplication, when the coefficients are optimized so that a setobjective function is minimized, a maximum value of the image signal ofthe input image is detected, and an image signal is used which isstandardized so that the maximum value of the image signal becomes 1.0by dividing each image signal of the input image by the maximum value.

[0029] Preferably, when each of the blackbody locus curve of skin colorand the blackbody locus curve of gray is set, a spectral sensitivity ofa photographing apparatus used to form the input color image is used asa spectral sensitivity distribution.

[0030] Preferably, when each of the blackbody locus curve of skin colorand the blackbody locus curve of gray is set, a spectral sensitivity ofBT709 is used as a spectral sensitivity distribution.

[0031] Moreover, in order to attain the first object described above,the second aspect of the present invention provides a white balancecorrection apparatus for correcting white balance when digital imageprocessing is performed on an input color image to form a print,comprising means for estimating, by using at least gray and/or skincolor information contained in the input color image, a colortemperature of a photographing light source with which the color imagehas been taken, and means for correcting image signals of the colorimage based on the estimated color temperature.

[0032] Preferably, the means for estimating a color temperature of thephotographing light source includes means for multiplying the imagesignals of respective pixels in the input color image by setcoefficients, skin color candidate pixel detection means for detectingpixels having image signals in the vicinity of a blackbody locus curveof skin color as a result of the multiplication and gray candidate pixeldetection means for detecting pixels having image signals in thevicinity of a blackbody locus curve of gray as a result of themultiplication, means for optimizing the coefficients so that the numberof the skin color candidate pixels and/or the number of the graycandidate pixels are maximized and a difference between an average colortemperature of the group of skin color candidate pixels and an averagetemperature of the group of gray candidate pixels is minimized, andmeans for computing a color temperature of the photographing lightsource from the average color temperature of the group of skin colorcandidate pixels and the average color temperature of the group of graycandidate pixels, and wherein the means for correcting an image signalof the color image is means for correcting the color image signalsmultiplied by the optimized coefficients by a difference between theestimated color temperature and a color temperature of reference white.

[0033] Preferably, the means for estimating a color temperature of thephotographing light source includes coefficient multiplication means formultiplying the image signals of respective pixels in the input colorimage by set coefficients, skin color candidate pixel detection meansfor detecting pixels having the multiplied image signals in the vicinityof a blackbody locus curve of skin color as a result of themultiplication, and gray candidate pixel detection means for detectingpixels having the multiplied image signals in the vicinity of ablackbody locus curve of gray as a result of the multiplication,optimization means for optimizing the set coefficients so that thenumber of the skin color candidate pixels and the number of the graycandidate pixels are maximized and a difference between an average colortemperature of the skin color candidate pixels and an average colortemperature of the gray candidate pixels is minimized, and colortemperature estimating and computing means for estimating the colortemperature of the photographing light source for a group of skin colorcandidate pixels and a group of gray candidate pixels obtained byoptimizing the set coefficients by the optimization means for optimizingthe set coefficients in which the group of skin color candidate pixelsis divided into a plurality of subgroups of skin color candidate pixelsand the group of gray candidate pixels is divided into a plurality ofsubgroups of gray candidate pixels, and a color temperature of thephotographing light source is estimated from an average colortemperature of a subgroup of skin color candidate pixels with a highaverage color temperature among the plurality of subgroups of skin colorcandidate pixels and an average color temperature of a subgroup of graycandidate pixels with a high average color temperature among theplurality of subgroups of gray candidate pixels, and wherein the meansfor correcting the image signals of the color image is means forcorrecting the image signals of the color image multiplied by theoptimized set coefficients by an amount corresponding to a differencebetween the estimated color temperature and a color temperature ofreference white.

[0034] Further, in order to attain the second object described above,the third aspect of the present invention provides a density correctionmethod, comprising the steps of multiplying image signals of respectivepixels in an input color image by set coefficients to detect pixelshaving the multiplied image signals in the vicinity of a blackbody locuscurve of skin color as skin color candidate pixels, and assigning anaverage obtained for predetermined color signals from the skin colorcandidate pixels detected to a predetermined density of a colorcorresponding to the color signals on a print.

[0035] Preferably, the predetermined color signals are G signals and anaverage G signal obtained from the skin color candidate pixels detectedis assigned to a predetermined C density on a print. Here, preferably,the predetermined G density is 0.7 to 1.0.

[0036] And, in order to attain the first and second objects describedabove, the fourth aspect of the present invention provides a recordingmedium on which one or both of a white balance correction methodaccording to each of the above-mentioned first aspect of the presentinvention and a density correction method according to each of theabove-mentioned third aspect of the present invention are recorded in acomputer-readable manner as a program to be executed by a computer.

BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In the accompanying drawings:

[0038]FIG. 1 is a block diagram schematically showing an embodiment of awhite balance correction apparatus in accordance with the presentinvention;

[0039]FIG. 2 is a chromaticity diagram for explaining the principle ofwhite balance correction in the embodiment;

[0040]FIG. 3 is a graph showing spectral sensitivity distributions of atypical CCD sensor;

[0041]FIG. 4 is a flowchart showing a flow of processing in theembodiment;

[0042]FIG. 5 is a diagram showing BT709 spectral sensitivitydistributions;

[0043]FIG. 6 is a block diagram schematically showing another embodimentof a white balance correction apparatus in accordance with the presentinvention; and

[0044]FIG. 7 is a flowchart showing a flow of processing in the secondembodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] The method and apparatus for correcting white balance and themethod for correcting density, as well as the recording medium on whicha program for carrying out these methods is recorded, all of which thepresent invention provides, will be described below in detail withreference to preferred embodiments of the invention shown in theaccompanying drawings.

[0046]FIG. 1 is a block diagram schematically showing an embodiment of awhite balance correction apparatus of a second aspect which performs awhite balance correction method of a first aspect in accordance with thepresent invention.

[0047] The white balance correction apparatus shown in FIG. 1 isarranged to correct white balance when digital image processing isperformed on input image signal to form a photographic print. Forexample, it is provided in an image processor in a digital photoprinteror the like.

[0048] The white balance correction apparatus indicated by 10 in FIG. 1is constituted mainly of a light source color temperature estimationdevice 12 for estimating, from an input color image, the colortemperature of a light source at the time of obtaining the color imageby photographing a subject, and an image signal correction device 14 formaking white balance correction by using the color temperature of thephotographing light source estimated by the light source colortemperature estimation device 12.

[0049] The light source color temperature estimation device 12 has acoefficient multiplication section 16, a skin color candidate detectionsection 18, a gray candidate detection section 20, a coefficientoptimization section 22, and a light source color temperaturecomputation section 24.

[0050] Before describing the functions of these devices and sections,the principle of the present invention will first be described.

[0051] While the conventional white balance correction method corrects acolor imbalance on the basis of the Evans' theorem so that the averageof colors through the entire frame is gray, the present invention ischaracterized by positively searching for a gray portion in a frame andby correcting an imbalance by performing exposure so that the graycandidate point is finished in gray on a print.

[0052] Here, the same white balance correction method is applied to animage forming process using a color negative film and to an imageforming process in a digital still camera (DSC). The principle of thepresent invention will be explained with respect to a case where a sceneis photographed with a DSC under ordinary conditions.

[0053] A case where a scene containing a gray portion (in gray or in acolor close to gray) is photographed with a DSC in natural daylight at acolor temperature of 4000 K will be considered as an example.

[0054] Image signals (R, G, B) of the photographed gray portion areplotted in a chromaticity diagram by being converted into chromaticitycoordinates (r, b) by the following equations (1):

r=R/(R+G+B)

b=B/(R+G+B)  (1)

[0055]FIG. 2 shows the chromaticity diagram. A curve Gy in FIG. 2 is ablackbody locus of gray. As is well known, a blackbody locus is formedin such a manner that if T is color temperature; a blackbody radiationenergy distribution at color temperature T is P(λ); a spectralreflectance distribution of a subject is ρ(λ); and a spectralsensitivity distribution of a CCD sensor is S_(i)(λ) (where i=B, G, R),E_(i) is computed by the following equation (2),

E _(i) =∫P(λ)ρ(λ)S _(i)(λ)dλ  (2)

[0056] and this E_(i) is plotted in a chromaticity diagram by beingconverted into chromaticity coordinates (r, b) by the equation (1) shownabove while color temperature T is changed.

[0057] A blackbody locus exists with respect to each of spectraldistributions of a CCD sensor and each of the colors of a subject, and ablackbody locus of gray is obtained by setting the spectral reflectanceρ(λ) of a subject in gray at 1. FIG. 3 shows spectral sensitivitydistributions of a typical CCD sensor. It is preferable to use suchS_(i)(λ) characteristic of a CCD sensor. However, BT709 ideal spectralsensitivity distributions such as shown in FIG. 5 may alternatively beused.

[0058] If the above-mentioned image signal of a gray portion is plottedin the chromaticity diagram shown in FIG. 2, it is considered to bescattered in the vicinity Gy0 of the point at 4000 K on the blackbodylocus of gray Gy.

[0059] In recent ordinary DSCs having an AWB (auto white balance)function, the gray portion is scattered in the vicinity Gy1 of astandard white (e.g., 5500 K) if the AWB function is suitably performed.However, if the AWB function is not suitably performed, the gray portionis scattered in a region at an unknown position remote from thephotographing temperature 4000 K (e.g., a region indicated by symbol Ain FIG. 2).

[0060] To convert the gray portion at such an unknown position indicatedby symbol A in FIG. 2 as included in the DSC image to the vicinity Gy1of a reference white (e.g., 5500 K), white balance correction is made ina printer. It is possible to expect that a well-gray-balanced good printwill be obtained by performing this conversion with respect to each ofthe pixels of the image.

[0061] Since the position of the region A shown in FIG. 2 is unknown, itis impossible to make a direct conversion from the region A to thevicinity Gy1 of a reference white (e.g., 5500 K). Therefore, in thepresent invention, the desired conversion is performed in two steps.

[0062] That is, conversions expressed by two vectors α and β shown inFIG. 2 are made.

[0063] The vector α is a vector for correction of the amount ofdeviation from the blackbody locus Gy due to the imperfection of the AWBfunction of the USC. The conversion using the vector a is performed as aconversion from the region A shown in FIG. 2 to the region Gy0 on theblackbody locus Gy.

[0064] The vector β is a vector for conversion from the region Gy0 tothe vicinity Gy1 of the reference white (e.g., 5500 K) along theblackbody locus Gy.

[0065] The conversion from the region A to the vicinity Gy1 of thereference white (e.g., 5500 K) as shown in FIG. 2 can be achieved bycombining the two vectors α and β.

[0066] The description will again be made of the devices and sectionsshown in FIG. 1.

[0067] It is difficult to obtain vector a expressing the conversion fromthe region A shown in FIG. 2 to the region Gy0 on the blackbody locus Gyin the above-described two conversion steps using vectors α and β. Ifthe region Gy0 can be obtained, the color temperature T of thephotographing light source can be estimated therefrom. Thus, it is easyto obtain the vector β expressing the conversion from Gy0 (colortemperature T) to Gy1 (color temperature of 5500 K) along the blackbodylocus Gy.

[0068] The light source color temperature estimation device 12 obtainsthe vector a to make the corresponding conversion, and the image signalcorrection device 14 obtains the vector β to make the correspondingconversion.

[0069] The AWB function of the DSC is performed in such a manner thatthe image signals R, G, and B of all the pixels, which are collectedimmediately after photographing, are uniformly multiplied by constants.This multiplication using constants is a linear transformation. If theimage signals have deviated from the blackbody locus by this operation,the process of making the gray portion scattered mainly at temperaturesin the vicinity (Gy0 in FIG. 2) of the color temperature of thephotographing light source on the blackbody locus (4000 K in this case)must be the operation having the effect reverse to that of the operationperformed as the AWB function.

[0070] The transformation which is the reverse of the above-describedAWB function (linear transformation) of the DSC is also a lineartransformation. Therefore the DSC image signals R, G and B aremultiplied by predetermined coefficients (in the process shown in FIG.2, the image signals R, G, and B are converted into chromaticitycoordinates). The coefficient multiplication section 16 performs thismultiplication.

[0071] The coefficient multiplication section 16 respectively multipliesR and G in the DSC image signals R, G and B by predeterminedcoefficients α₁ and α₂ as shown by the following equations (3) toconvert R and G into R′ and G′.

R′=α₁R

G′=α₂G  (3)

[0072] For conversion in the coefficient multiplication section 16, itis not necessary to change the three signals; changing two of thesignals may suffice.

[0073] The region A to which the point corresponding to the originalgray has been moved is unknown and it is impossible to identify theportion corresponding to the original gray. Therefore the gray candidatedetection section 20 compares the signal converted by the lineartransformation with the blackbody locus of gray, recognizes pixelsdetected in the vicinity of the blackbody locus as highly probable tocorrespond to the original gray, and sets the pixels as gray candidatepixels. Determination as to whether pixels are in the vicinity of theblackbody locus may be made according to whether the distance in termsof chromaticity coordinates (r, b) from the blackbody locus is withinthe range of 0.01.

[0074] The coefficient optimization section 22 counts the number of graycandidate pixels detected by the gray candidate detection section 20,and optimizes the coefficients α₁ and α₂ by making the coefficientmultiplication section 16 and the gray candidate detection section 20perform the above-described operations while changing the predeterminedcoefficients α₁ and α₂ so that the number of gray candidate pixels ismaximized.

[0075] A suitable method, not particularly specified, may be used forthis optimization. For example, a simplex method, which is a standardmethod for numerical value computation, is preferably used. Thecoefficients α₁ and α₂ in the equations (3) for linear transformationare obtained by the above-described optimization as the components ofvector a corresponding to the operation reverse to the AWB function ofthe DSC. That is, α=(α₁, α₂).

[0076] To improve the accuracy of optimization, information on a colorother than gray may also be used. Skin color can be selected fromsubject colors as a color appearing frequently in photographing andcomparatively stable in color tint among its variations. Skin color isthought to vary considerably with respect to human races (white race,yellow-skinned race, and black race). However, it is known throughspectrometric analysis that skin color varies mainly in lightness amongraces and does not vary largely in spectral composition, that is, skincolor has only small variation in the color tint. This characteristic ofskin color can be utilized for color identification.

[0077] Therefore the skin color candidate detection section 18 also setsa blackbody locus (not shown) with respect to skin color as well as thatof gray, and detects skin color candidate pixels having a color close tothe blackbody locus of skin color from the image signals multiplied bythe coefficients in the coefficient multiplication section 16. Notethat, here, as the skin color used in the computation of the blackbodylocus of skin color, for example, light skin of the Macbeth chart can beused. However, the present invention is not limited to this color.

[0078] The coefficient optimization section 22 also counts the number ofskin color candidate pixels detected by the skin color candidatedetection section 18, combines it with the number of gray candidatepixels described above, and optimizes the coefficients α₁ and α₂ SO thatthe number of these pixels is maximized, thus improving the accuracywith which the coefficients α₁ and α₂ are optimized. Note that, in thepresent invention, if primary importance is attached to the skin color,the coefficients α₁ and α₂ may be optimized so that the number of onlythe skin color candidate pixels detected by the skin color candidatedetection section 18 is maximized.

[0079] It is also expected that it a scene including a gray portion anda skin color portion is photographed with a uniform light source, theaverage color temperatures of the group of gray candidate pixels and thegroup of skin color candidate pixels, detected as pixels having colorsclose to the blackbody locus, coincide with each other. In optimizationin the coefficient optimization section 22, therefore, optimization ofthe above-described coefficients may be performed by setting anobjective function for “minimizing the difference between the averagecolor temperatures of the group of gray candidate pixels and the groupof skin color candidate pixels”.

[0080] Further, the two above-described methods may be combined tooptimize the above-described coefficients by setting the objectivefunction for “maximizing the number of at least one of gray candidatepixels and skin color candidate pixels” and the objective function for“minimizing the difference between the average color temperatures of thegroup of gray candidate pixels and the group of skin color candidatepixels”.

[0081] The method using these two objective functions further improvesthe optimization accuracy.

[0082] Next, the light source color temperature computation section 24converts the image signals by using the coefficients α₁ and α₂ optimizedin the above-described coefficient optimization section 22, computes theaverage color temperature Tg of the group of gray candidate pixels onthe blackbody locus of gray and/or the average color temperature Tf ofthe group of skin color candidate pixels on the blackbody locus of skincolor, and computes the color temperature T of the photographing lightsource from these average color temperatures. The color temperature Tmay be computed by averaging as shown by T=(Tg+Tf)/2 or may be obtainedas T=Tg if only gray is present or if gray is more important than skincolor. It may also be obtained as T=Tf if only the skin color is presentor if the skin color is more important than gray. The color temperatureT of the photographing light source is thus estimated.

[0083] Further, since it can be considered that a photographing lightsource for photographing a subject to obtain a color image is usuallyone in the above-described example, the number of photographing lightsources for which a color temperature is estimated is one, and anappropriate photographing source is estimated. However, if a subject isphotographed indoors using a strobe (e.g., under a tungsten lamp), thisprecondition may not be realized.

[0084] That is, one of the points that should be taken into account hereis handling of an image photographed with strobe light, in particular,an image photographed indoors with strobe light, for example, underpresence of other light sources such as a tungsten light source. In thiscase, two light sources with completely different color temperaturescoexist. That is, light in a part illuminated by the strobe light (e.g.,a person in the center of a frame image) has a color temperature in theorder of 5000 K, which is a color temperature of strobe light, and lightin a part not illuminated by the strobe light at the rim of the imagephotographed with strobe light (illuminated by the light of the tungstenlamp) has a color temperature in the order of 3000 K, which is a colortemperature of light of a tungsten lamp.

[0085] As a result, a color temperature of a photographing light sourcein the case in which the number of photographing light sources isestimated to be one as in the above-described example becomes an averageof both the color temperatures, for example, 4000 K, and is estimated tobe lower than 5000 K of the color temperature of the strobe that is amain photographing light source. Thus, the skin color of the person, whois the subject, is tinged blue. Consequently, consideration for thisproblem is required.

[0086] Therefore, in another embodiment of the present invention,ingenuity is put into the method of computing a color temperature of alight source in order to improve white balance correction of an imagephotographed with strobe light involving such light sources of differenttypes.

[0087] In this embodiment, the light source color temperaturecomputation section 24 converts an image signal using the optimizedcoefficients α₁ and α₂ to compute an average color temperature Tg of agroup of gray candidate pixels on a blackbody locus of gray and anaverage color temperature Tf of a group of skin color candidate pixelson a blackbody locus of skin color as in the above-described example. Inthis case, the light source color temperature computation section 24 inaccordance with this embodiment divides the above-mentioned group ofgray candidate pixels and group of skin color candidate pixels into twosubgroups corresponding to different light sources (a tungsten lightsource and a strobe light source in this context), respectively, (twosubgroups of gray candidate pixels and two subgroups of gray candidatepixels). The light source color temperature computation section 24 thenconverts, for example, the number of candidate pixels belonging to eachsubgroup into a histogram and determines that the subgroup with thelarger number of the candidate pixels is a principal (main) lightsource. That is, when the group of gray candidate pixels and the groupof skin color candidate pixels are converted into a histogram,respectively, a plurality of peaks (two peaks in the above-describedexample) appear. Thus, the light source color temperature computationsection 24 divides the groups having each peak into subgroups,respectively, and determines that the subgroup with the higher peak,that is, the subgroup with the larger number of candidate pixels is theprincipal light source.

[0088] Then, the light source color temperature computation section 24finds average temperatures (Tf, Tg) of each subgroup that is determinedas the principal light source and computes a color temperature T of aphotographing light source. In this computation of the color temperatureT of the photographing light source, for example, it is sufficient toconsider that T=Tf if only the skin color is present or if the skincolor is more important than gray and consider that T=Tg if only gray ispresent or if gray is more important than the skin color.

[0089] In this way, the color temperature T of the photographing lightsource is estimated.

[0090] Finally, the image signal correction device 14 makes a conversionfrom the thus estimated color temperature T to the reference white(e.g., 5500 K) along the blackbody locus. This conversion can beexpressed as linear transformation of the R and B signals shown by thefollowing equations:

R″=β₁R′

B″=β₂B  (4)

[0091] These coefficients β₁ and β₂ are obtained as the components ofvector β.

[0092] The image signal correction device 14 performs conversion usingthis vector β on each pixel to complete white balance correction withrespect to each pixel.

[0093] The above-described conversion process is summarized below. Theconversion from the point A to the reference white (e g., 5500 K) Gy1 asshown in FIG. 2 is realized as a resultant of transformation by vector aand transformation by vector β, as shown by the following equations (5):

R″=α₁β₁R

G″=α₂G

B″=β₂B  (5)

[0094] The equations (5) include a change in lightness as well as acolor balance. These equations may be rewritten as shown below byassuming that the G signal is constant.

R″=(α₁β₁/α₂)R

G″=G

B″=(β₂/α₂)B  (6)

[0095] The operation of this embodiment will be described with referenceto the flowchart of FIG. 4.

[0096] First, in step 100, a scene is photographed in light from acertain light source by using a digital still camera (DSC).

[0097] In step 110, the image signals R, G, and B of the image formed bythe DSC are input.

[0098] In step 120, image signal optimization processing is performed bythe coefficient multiplication section 16, the skin color candidatedetection section 18, the gray candidate detection section 20 and thecoefficient optimization section 22 of the light source colortemperature estimation device 12. This processing is intended to restorethe image signals, which are caused to deviate from a blackbody locus bythe AWB function of the DSC, to signals (raw data) without a deviationin the vicinity of the blackbody locus. The coefficient multiplicationsection 16 multiplies all the image signals by predeterminedcoefficients. The skin color candidate detection section 18 comparesthis data with the blackbody locus of skin color to detect data (skincolor candidate pixels) considered to have a color in the vicinity ofthe skin color on the blackbody locus. The gray candidate detectionsection 20 compares this data with the blackbody locus of gray to detectdata (gray candidate pixels) considered to have a color in the vicinityof the gray on the blackbody locus.

[0099] The coefficient optimization section 22 counts the detected skincolor candidate pixels and/or the detected gray candidate pixels, andoptimizes the above-described coefficients by resetting the coefficientsand repeating setting the coefficients so that the number of the skincolor candidate pixels and/or gray candidate pixels is maximized or thedifference between the average color temperature of the group of skincolor candidate pixels and the average color temperature of the group ofgray candidate pixels is minimized, or both of these conditions aresatisfied.

[0100] In step 130, the light source color temperature computationsection 24 computes the estimated value T of the color temperature ofthe photographing light source from the average color temperature Tf ofthe group of skin color candidate pixels and the average colortemperature Tg of the group of gray candidate pixels, obtained bymultiplication by the optimized coefficients. Note that, if illuminationis considered to be formed of a plurality of light sources, for example,two light sources, it is preferable to compute an estimated value T of acolor temperature of the light sources taking into account a result ofsub-grouping described above.

[0101] Next, in step 140, the image signal correction device 14determines the amount of correction from the light source colortemperature T estimated in the light source color temperaturecomputation section 24 to the reference white (e.g., 5500 K), andcorrects the white balance of all the pixels by this amount ofcorrection.

[0102] Finally, in step 150, other kinds of image processing areperformed on the image signals, and a finished print is output from theprinter.

[0103] As described above, according to the photoprinter incorporatingthe white balance correction apparatus in accordance with the presentinvention, an effect of improvement in a white balance correctioncapability on a finished print is recognized compared with the case ofprinting by the conventional technique.

[0104] Here, a supplementary explanation will be given on step 130 ofFIG. 4 in the case in which two-light source illumination is used.

[0105] A group of gray candidate pixels and a group of skin colorcandidate pixels that are detected as near colors of a blackbody locuscurve are distributed dispersing (separating) in two parts near a colortemperature of an incandescent lamp (3000 K) and near a colortemperature of strobe light (5000 K to 5500 K). In this embodiment,these pixel groups are not simply averaged for the color temperaturesbut a histogram (cumulative frequency distribution) is created withrespect to the color temperatures.

[0106] Then, for skin color, the group of skin color candidate pixelsare divided into a first subgroup of skin color candidate pixels and asecond subgroup of skin color candidate pixels with an inflection pointof a skin color cumulative frequency distribution as a boundary to findan average color temperature of each subgroup. Then, a higher averagecolor temperature is regarded as an average color temperature of theentire group of skin color candidate pixels. For gray, in the samemanner, the group of gray candidate pixels are divided into a firstsubgroup of gray candidate pixels and a second subgroup of graycandidate pixels with an inflection point of a gray cumulative frequencydistribution as a boundary to find an average color temperature of eachsubgroup. Then, a higher average color temperature is regarded as anaverage color temperature of the entire group of gray candidate pixels.A color temperature of the photographing light source is estimatedaccording to an average value of the average color temperature of thegroup of skin color candidate pixels and the average color temperatureof the group of gray candidate pixels.

[0107] Note that if the inflection point of the cumulative frequencydistribution cannot be found, it is sufficient to regard thatsingle-light source illumination is used and estimate a colortemperature by the method in the case of using the above-describedsingle-light source illumination.

[0108] Further, in the present invention, for example, when imagesignals of respective pixels in an input color image are multiplied byset coefficients in the coefficient multiplication section 16 and, as aresult, when the coefficient is optimized in the coefficientoptimization section 22 so that the above-described predeterminedobjective function is minimized, it is preferable to detect a maximumvalue of the image signal of the input image and divide each imagesignal of the input image by this maximum value, thereby using an imagesignal that is standardized so that the maximum value of the imagesignal becomes 1.0. In this way, the present invention can be applied toan image even if it is photographed in a state of considerable lack ofexposure and/or excessive exposure (underexposure/overexposure).

[0109] The present invention will be further described with respect toconcrete examples thereof.

EXAMPLE 1

[0110] The performance of the above-described white balance correctionmethod was tested when prints were formed from 309 frames of DSC imagesobtained by picture-taking with typical two DSC models from Fuji PhotoFilm Co., Ltd. For comparison with the prints formed in accordance withthe present invention, original images (prints which were output withoutcorrection after AWB in the DSCs) and prints which were obtained byperforming on the images from the DSCs white balance correction based onthe conventional art were prepared. The prints evaluated were sortedinto 3 groups; good ones (X), unsatisfactory ones (Y), and defectiveones (Z). Only the good ones (X) were counted as passed items.

[0111] First, 8-bit DSC image signals R, G, and B were converted intosubject-linear signals R₀, G₀, and B₀, as described below.

[0112] 8-bit DSC image signals R, G, and B obtained by picture-takingwith the DSC are processed as described below. First, subject-linearsignals R_(o), G_(o), and B_(o) generated from the CCD sensor areconverted by gamma-0.45 nonlinear transformation shown by the followingequations (7):

R ₁=1.099×R _(o) ^(0.45)−0.099

G ₁=1.099×G _(o) ^(0.45)−0.099

B ₁=1.099×B _(o) ^(0.45)=0.099  (7)

[0113] Thereafter, color difference signals Y₁, C_(r1) and C_(b1) areformed therefrom as shown by the following equations (8):

Y ₁=0.30R ₁+0.59G ₁+0.11B ₁

C _(r1)=0.70R ₁=0.59G ₁−0.11B ₁

C _(b1)=−0.30R ₁−0.59G ₁+0.89B ₁  (8)

[0114] A color-difference matrix operation is performed on these signalsto improve the chromaticity, thereby making conversions intocolor-difference signals Y₂, C_(r2), and C_(b2) shown by the followingequations (9):

Y₂=Y₁

C _(r2)=1.625C _(r1)=0.2734C _(b1)

C _(b2)=−0.08203C _(r1)+1.6094C _(b1)  (9)

[0115] Finally, R, G and B signals are restored and converted into 8-bitsignals by the following equations (10) to obtain 8-bit DSC imagesignals R, G, and B.

R=Y ₂ +C _(r2)

G=Y ₂−0.51C _(r2)−0.18C _(b2)

B=Y ₂ +C _(b2)  (10)

[0116] To convert 8-bit DSC image signals R, G, and B intosubject-linear signals R₀, G₀, and B₀, therefore, the inverse operationprocess starting from processing R, G, and B signals and the operationsin the order from equations (10), equations (9), equations (8), andequations (7) may be performed. This inverse operation process wasperformed to obtain the subject-linear signals R₀, G₀, and B₀.

[0117] Next, the amounts of white balance correction based onoptimization of subject-linear signals R₀, G₀, and B₀ were computed.

[0118] To enable this computation, the blackbody locus of gray and theblackbody locus of skin color were formed in advance by using thespectral sensitivity of the DSC used for picture-taking. Operations foroptimizing the R_(o), G_(o), and B_(o) signals were performed by usingthese loci to obtain vector a and vector β, and white balance correctionsignals R″, G″, and B″ were obtained as shown by equations (11) below.This optimization computation was performed by optimizing thecoefficients so that the number of gray candidate pixels and skin colorcandidate pixels was maximized and the difference between the averagecolor temperature of the group of skin color candidate pixels and theaverage color temperature of the group of gray candidate pixels wasminimized.

R″=(α₁β₁/α₂)R ₀

G″=G₀

B″=(β₂/α₂)B ₀  (11)

[0119] To convert these white balance correction signals R″, G″, and B″into 8-bit image signals, the operations may successively be performedin the order of equations (7), equations (8), equations (9), andequations (10). These signals were output from the printer to obtain awhite-balance-corrected print.

[0120] Table 1 shows the results of comparison between the prints afterwhite balance correction in accordance with the present invention, theoriginal images and those obtained by the conventional art. TABLE 1Original Conventional Present Images art Invention Passing Rate (%) 65.373.3 90.0

[0121] As shown in Table 1, the passing rate of the present inventionwas higher by about 12 points than that of the conventional art, and thehigh white balance correction performance of the present invention wasconfirmed.

EXAMPLE 2

[0122] In Example 1, the names of DSCs were known and the spectralsensitivity and the color processing algorithm were also known. However,if the white balance correction method of the present invention is usedas printing software, it is desirable to ensure applicability tocorrection of DSC images formed by unknown models (or to ensurerobustness).

[0123] In Example 2, a white balance correction test was made withrespect to 309 frames of DSC images obtained by picture-taking withtypical two DSC models from Fuji Photo Film Co., Ltd, which are same asthose used in Example 1, and 240 frames of images obtained bypicture-taking of the same scenes (16 frames) with fifteen unidentifiedmodels from other manufactures.

[0124] In this example, white balance correction in accordance with thepresent invention was performed under the assumption that all modelswere DSCs having ideal spectral sensitivities, because, even if themodel cannot be identified, the performance of any model can beapproximated to the ideal spectral sensitivity characteristics of BT709as shown in FIG. 5 by virtue of the combination of the spectralsensitivity and the color processing algorithm.

[0125] That is, in this example, the spectral sensitivity distributionsof BT709 shown in FIG. 5 were used as spectral sensitivity distributionS_(i) (i=R, G, B) of the CCD sensor in equation (2) for obtaining ablackbody locus.

[0126] In this example, since DSCs having the ideal spectralsensitivities do not need the color processing algorithm for improvingthe chromaticity, it is not necessary to perform, for conversion of8-bit DSC image signals R, G, and B into subject-linear signals R_(o),G_(o), and B_(o), the inverse operation process from equations (10) toequations (7) required in Example 1, and the inverse operation ofequations (7) can be immediately performed.

[0127] The blackbody locus of gray and the blackbody locus of skin colorwere formed in advance by using the ideal spectral sensitivity of BT709,the operations for optimizing R₀, G₀, and B₀ signals were performed byusing these loci to obtain vector a and vector β, as in Example 1. Whitebalance correction signals R″, G″, and B″ expressed by equations (11)were obtained therefrom.

[0128] These signals were converted by gamma-0.45 nonlinear conversionand converted signals were 8-bit quantized to obtain 8-bit imagesignals, which were supplied to a printer to obtain awhite-balance-corrected print.

[0129] The prints thus obtained were evaluated, as were those inExample 1. The prints evaluated were sorted into good ones (X),unsatisfactory ones (Y), and defective ones (Z). Only the good ones werecounted as passed items. Table 2 shows the evaluation results. TABLE 2Typical models from Fuji Photo Film Co., Ltd. 16 scenes (240 frames) by15 (309 frames) models from other manufacturers Present OriginalConventional Present Invention images art invention Passing 88.0 47.170.0 76.2 rate (%)

[0130] As shown in Table 2, substantially the same passing rate (88%) asthat in Example 1, though slightly lower, was maintained with respect tothe two typical models from Fuji Photo Film Co., Ltd. (309 frames). Withrespect to the fifteen models from the other manufacturers (240 frames),it is observed that the absolute value of the passing rate achieved bythe present invention was low (76%) than that achieved by the twotypical models from Fuji Photo Film Co., Ltd. but higher by about 6points than that achieved by the conventional art.

[0131] The reason for the low passing rate may be because, as can beunderstood from the considerably low passing rate (47%) of the originalimages, the number of evaluated scenes was small (16 scenes) and theimages had imbalance details, and also because the actual DSCs wereassumed to be DSCs having ideal spectral sensitivities.

[0132] As described above, according to Example 2, the white balancecorrection method of the present invention is sufficiently effectivewhen provided as common software and achieves a sufficiently high whitebalance correction effect even with respect to image forming apparatusmodels whose spectral sensitivity and color processing algorithm areunknown.

[0133] Further, in the above-mentioned examples, in the optimizationcomputation, a white balance correction signal is found by optimizingcoefficients so that the number of gray candidate pixels Ng and thenumber of skin color candidate pixels Nf are maximized and thedifference between the average color temperature Tg of the group of graycandidate pixels and the average color temperature Tf of the group ofskin color candidate pixels is minimized. In this case, there is oneobjective function F for optimization, which is represented by thefollowing equation:

F=abs(Tg−Tf)−(Ng+Nf)

[0134] Here, abs indicates an absolute value. Since the simplex methodworks so that a set objective function is minimized, if maximization ofthe number of candidate pixels is desired, it is sufficient to make(Ng+Nf) subtracted as shown above.

[0135] Even if there is only one objective function, a favorable resultwas obtained as a whole as described above. However, looking at theprints in detail, defective prints (prints of the evaluation Z(defective ones)) were found here and there. Since it was important toreduce the number of prints of the evaluation Z (defective ones) as muchas possible in the automatic printing work, an effect in the case of anincreased number of objective functions was investigated. The number ofobjective functions was increased to two. The following equation wasused as a second objective function:

F*=abs(Tg−Tf)−Ng

[0136] A first white balance correction signal obtained by optimizingcoefficients using the first objective function and a second whitebalance correction signal obtained by optimizing coefficients using thesecond objective function were averaged to obtain a new white balancecorrection signal to form prints. Then, approximately a half of theprints that were given the evaluation Z (defective ones) changed toprints of the evaluation Y (unsatisfactory ones). The number of printsof the evaluation X (good ones) hardly changed, and as a whole, thenumber of prints of the evaluation Y (unsatisfactory ones) increased andthe number of prints of the evaluation Z (defective ones) decreased. Asa result, stable prints were obtained. Moreover, the combination of twoobjective functions is not limited to this. When other combinations werechecked, a combination of the following two objective functions weremost effective for stabilization of prints:

F=abs(Tg−Tf)−Ng

F*=abs(Tg−Tf)−Nf

[0137] A white balance correction method of the embodiment in this casewill be described with reference to FIG. 6.

[0138] As shown in FIG. 6, a first image signal after white balancecorrection corresponding to the first objective function, which isoutputted from a white balance correction apparatus 10, is saved in amemory 30. Subsequently, a second image signal after white balancecorrection corresponding to the second objective function is saved in amemory 31. Thereafter, a new image signal after white balance correctionis generated from the first and second image signals after white balancecorrection and used as a signal for forming prints.

[0139] Moreover, as a method of stabilizing print outputs, it was alsofound to be effective to form prints with weakened correction ratherthan directly using an image signal after gray balance correctionobtained by an optimization computation. A degree of weakening ispreferably 60 to 80%. In addition, correction may be weakened to 60 to80% according to a BV value (index indicating brightness of an image)written in an Exif file of a DSC image.

[0140] In addition, in the optimization computation described above, apixel with a small signal value (dark pixel) was considered to havelittle effective information in the past. Thus, for saving a computationtime, a pixel with a signal value equal to or less than a lower limitvalue (0.08) was not used in the computation. However, in anunderexposed image, since signal values of many pixels become lower thanthe lower limit value of 0.08, the number of pixels that can be used inthe optimization computation considerably decreases and computationaccuracy falls. To the contrary, in an overexposed image, for example,in the case in which a white wall is illuminated by a tungsten lamp andexposure is appropriate, all signal values of R=1.0, G=0.7, B=0.5 of asignal are clipped to 1.0 such as R=1.0, G=1.0, B=1.0, and the signalturns into a signal just like that of reference light sourceillumination. However, in the case in which a signal value is 1.0, it isdifficult to determine whether it is a real value or a value caused byclipping. Therefore, it is necessary to examine pixels to be used forthe optimization computation depending on underexposure or overexposure.For example, it is possible to lower a lower limit value and increasethe number of pixels. But it is bothersome to change a set value foreach image. Thus, in order to automatically examine pixels, an imagesignal is standardized so that its maximum value becomes 1.0 to allowthe number of pixels that can be used in the optimization computation tobe maintained substantially constant regardless of underexposure oroverexposure as described below.

[0141] That is, pixels in which at least one of R, C, and B signals is1.0 are excluded, and maximum values (R_(max), G_(max) and B_(max)) arefound for R, G and B signals, respectively, for all the remainingpixels. A maximum value and a minimum value among the maximum values areassumed to be T_(max) and T_(min), and R, G and B signals of an imageare standardized by T_(max). Consequently, an image just like an imagephotographed by appropriate exposure is obtained. A range of signal useis limited to a bright range of 1.0 to 0.25×(T_(min)/T_(max)), wherebyappropriate white balance correction can be realized for any images ofappropriate underexposure and overexposure without using so many pixelsin the optimization computation. In a scene of snow under overexposure,whiteness of the snow was successfully detected and beautiful finishprints could be realized by an effect of the standardization processing.

[0142] In the present embodiment of the present invention, as describedabove in detail, an algorithm using only gray and/or skin colorinformation in a DSC image is constructed to correct white balance atthe time of printing, and the apparatus and method of the presentinvention are advantageously effective in correcting white balance incomparison with the conventional art, as can also be understood from theexamples. The effect of determining whether a color tint of the whole ofan image is due to the photographing light source or the subject inaccordance with the present invention, is advantageously improved incomparison with the conventional art having an imperfection with respectto this effect. In particular, the correction performance of the presentinvention is substantially perfect with respect to a high colortemperature of a shaded scene or a scene under cloudy weather (7000 to10,000 K), so that while an entirely bluish print in which a human facehas color subsidence is formed in such a situation by the conventionalart, a print in which revived white is exhibited and in which skin coloris natural can be obtained according to the present invention.

[0143] In addition, as described above, all signals of an image arestandardized so that a signal value of a brightest pixel in the imagebecomes 1.0, whereby it becomes possible to perform the optimizationcomputation and process the image regardless of underexposure oroverexposure. Consequently, an image of the same quality as an imagephotographed by appropriate exposure can be obtained.

[0144] Next, the method for correcting density as a second embodiment ofthe present invention is described with reference to the flowchart ofFIG. 7. If the type of a light source is unknown, according to thismethod, skin color (not limited specifically to that of a face) in animage is detected utilizing the method for detecting skin colorcandidate pixels as stated above in the description of the firstembodiment of the invention, and then the print density is determinedbased on the information about the color to form a proper print.

[0145] Apparatus for effecting the second embodiment of the inventionare exemplified by a digital photoprinter comprising an image processorprovided with the coefficient multiplication section 16, the skin colorcandidate detection section 18 and the coefficient optimization section22 in the white balance correction apparatus 10 as stated above in thedescription of the first embodiment of the invention (as well as adensity correction section).

[0146] First in step 200, a scene is photographed in light from acertain light source by using a digital still camera (DSC) and then, instep 210, the image signals R, G and B of the formed image are input. Instep 220, the input signals are subjected to the skin color candidatedetection processing as stated above in the description of the firstembodiment of the invention, so that skin color candidate pixels aredetected. In particular, all the input image signals are multiplied bypredetermined coefficients and the data thus obtained are compared withthe blackbody locus of skin color to detect data, which are consideredto be in the vicinity of the skin color on the blackbody locus, as skincolor candidate pixels. At this time, the number of the detected skincolor candidate pixels may additionally be counted and the coefficientsfor the multiplication as above may be optimized so that the abovenumber is maximized, or so that the difference between the average colortemperature of a group of detected skin color candidate pixels and theaverage color temperature of a group of gray candidate pixels obtainedin a similar way to the skin color candidate pixels is minimized, or sothat these two conditions are both satisfied, to perform themultiplication using the coefficients thus optimized and obtain skincolor candidate pixels.

[0147] In the next step 230, density correction is performed. Inparticular, the average of the color signals (R, G and B) of the skincolor candidate pixels detected as stated above is initially determined.For this purpose, it may be available the average of the color signalsR, G and B((R+G+B)/3), or any specific color signal, G signal, forexample. The signal to be used is not particularly limited, although itis preferable to use G signal.

[0148] In the case of using G signal, density correction is performed byassigning the average obtained for G signal to, a predetermined Gdensity D (D=0.7, for example) on a print. The G density D is preferablybetween 0.7 and 1.0 both inclusive.

[0149] In step 240, the data obtained at the end of density correctionare output by the printer.

[0150] In this way, even if the type of a photographing light source isunknown, the density of a human face as a principal subject will be madeproper by detecting skin color and performing density correction basedon the information about the color. In consequence, a print can beproperly finished.

[0151] As an example, density correction was performed with respect toan image formed by photographing with a DSC a backlighted scene (a scenewhere a person in the center is backlighted), which often fails to beproperly photographed with a DSC under density correction by LATDmethod, and following results were obtained.

[0152] The print formed under density correction by LATD method wasevidently not appropriate, because, although the density through theprint was satisfactory, the face of the person looked pitch-dark.

[0153] In contrast, in the case of the print which was formed underdensity correction utilizing the detection of skin color according tothe second embodiment of the invention, assuming that the type of alight source is unknown, the face of the person had a proper density,even though the background density was rather lower, the print beingsubstantially satisfactory. This effected by virtue of the fact thatskin color was successfully detected and printing was performed underdensity correction based on such a detection of skin color. Proper printdensities could also he achieved with respect to other scenes than whatis referred to above.

[0154] As described above in detail, according to the above-mentionedeach embodiment, since an algorithm using only gray and/or skin colorinformation in a DSC image is established to perform white balancecorrection at the time of forming prints, a remarkably better correctioncapability can be obtained compared with the conventional technique.

[0155] Further, although a DSC image is described in the above-mentionedembodiment, the white balance correction method of the present inventioncan be applied not only to a DSC image but also to an image photographedon a color negative film.

[0156] If one or both of the white balance correction method and thedensity correction method as described hereinabove are recorded as acomputer-executable program on a computer-readable recording medium, thewhite balance correction method or the density correction method of thepresent invention may be carried out in a suitable apparatus such as animage processor by loading the program from the recording medium intothe apparatus employed.

[0157] The apparatus for correcting white balance, the method forcorrecting white balance and the method for correcting density, as wellas the recording medium on which a program for carrying out thesemethods is recorded, all of which the present invention provides, havebeen described in detail with various embodiments and examples. Needlessto say, the present invention is not limited to the describedembodiments and examples, and various modifications and changes of thedescribed embodiments and examples can be made without departing fromthe scope of the invention.

[0158] For example, in the above-mentioned embodiments, in theoptimization computation, coefficients are optimized so that the numberof gray candidate pixels Ng and the number of skin color candidatepixels Nf are maximized and the difference between the average colortemperature Tg of the group of gray candidate pixels and the averagecolor temperature Tf of the group of skin color candidate pixels isminimized. However, it is possible to change this to another objectivefunction.

[0159] In addition, as described above, it is possible to merchandisethe present invention as a program for making a computer execute theabove-mentioned white balance correction method and/or densitycorrection method and a recording medium having this programcomputer-readably recorded thereon. Moreover, it is also possible tocommercialize the present invention as a white balance correctionapparatus and/or a density correction apparatus that implements theabove-mentioned white balance correction method and/or densitycorrection method.

[0160] According to each aspects of the present invention, as describedabove, an algorithm is constructed to correct white balance byestimating the color temperature of the photographing light source usedin photographing a color image only from gray and/or skin colorinformation contained in the input color image, which enables suitablewhite balance correction at a high hit rate with respect to any inputimages regardless of DSC models used for forming images, or even in acase of photographing indoors with strobe light.

[0161] In particular, according to the present invention, in the case inwhich an entire image is standardized using a brightest pixel in theimage, the optimization computation can be performed to process theimage regardless of underexposure or overexposure, and an image ofappropriate exposure can be obtained.

[0162] Moreover, according to the present invention, by detecting skincolor in an image and performing density correction based on theinformation about the color, the print density can be made proper evenwith respect to those images of scenes where it is difficult with priorarts to make the print density proper.

What is claimed is:
 1. A white balance correction method, comprising thesteps of: estimating, by using at least gray and/or skin colorinformation contained in an input color image, a color temperature of aphotographing light source with which the color image has been taken;and correcting image signals of the color image based on the estimatedcolor temperature.
 2. The white balance correction method according toclaim 1, wherein only said gray and skin color information is used whencorrecting the image signals of the color image.
 3. The white balancecorrection method according to claim 1, wherein said estimating stepincludes: multiplying the image signals of respective pixels in theinput color image by set coefficients; setting pixels having themultiplied image signals in the vicinity of a blackbody locus curve ofskin color as skin color candidate pixels and/or pixels having themultiplied image signals in the vicinity of a blackbody locus curve ofgray as gray candidate pixels; optimizing the set coefficients so thatthe number of the skin color candidate pixels and/or the gray candidatepixels is maximized; obtaining a group of skin color candidate pixelsand/or a group of gray candidate pixels by using the optimized setcoefficients; and estimating the color temperature of the photographinglight source from an average color temperature of the group of skincolor candidate pixels and/or an average color temperature of the groupof gray candidate pixels, and wherein the image signals of the colorimage multiplied by the optimized set coefficients are corrected by anamount corresponding to a difference between the estimated colortemperature and a color temperature of reference white.
 4. The whitebalance correction method according to claim 3, wherein the colortemperature of the photographing light source is estimated from theaverage color temperature of the group of skin color candidate pixelsand the average color temperature of the group of gray candidate pixelsobtained by optimizing the set coefficients so that the number of theskin color candidate pixels and the gray candidate pixels is maximized.5. The white balance correction method according to claim 1, whereinsaid estimating step includes: multiplying the image signals ofrespective pixels in the input color image by set coefficients; settingpixels having the multiplied image signals in the vicinity of ablackbody locus curve of skin color as skin color candidate pixels andpixels having the multiplied image signals in the vicinity of ablackbody locus curve of gray as gray candidate pixels; optimizing theset coefficients so that a difference between an average colortemperature of the skin color candidate pixels and an average colortemperature of the gray candidate pixels is minimized; obtaining a groupof skin color candidate pixels and a group of gray candidate pixels byusing the optimized set coefficients; and estimating the colortemperature of the photographing light source from the average colortemperature of the group of skin color candidate pixels and the averagecolor temperature of the group of gray candidate pixels, and wherein theimage signals of the color image multiplied by the optimized setcoefficients are corrected by an amount corresponding to a differencebetween the estimated color temperature and a color temperature ofreference white.
 6. The white balance correction method according toclaim 1, wherein said estimating step includes: multiplying the imagesignals of respective pixels in the input color image by setcoefficients; setting pixels having the multiplied image signals in thevicinity of a blackbody locus curve of skin color as skin colorcandidate pixels and pixels having the multiplied image signals in thevicinity of a blackbody locus curve of gray as gray candidate pixels;optimizing the set coefficients so that the number of the skin colorcandidate pixels and/or the gray candidate pixels is maximized and adifference between an average color temperature of the skin colorcandidate pixels and an average color temperature of the gray candidatepixels is minimized; obtaining a group of skin color candidate pixelsand a group of gray candidate pixels by using the optimizedcoefficients; and estimating the color temperature of the photographinglight source from the average color temperatures of the group of skincolor candidate pixels and the group of gray candidate pixels, andwherein the image signals of the color image multiplied by the optimizedset coefficients are corrected by an amount corresponding to adifference between the estimated color temperature and a colortemperature of reference white.
 7. The white balance correction methodaccording to claim 1, wherein said estimating step includes: multiplyingthe image signals of respective pixels in the input color image by setcoefficients; setting pixels having the multiplied image signals in thevicinity of blackbody locus curve of skin color as skin color candidatepixels and pixels having the multiplied image signals in the vicinity ofa blackbody locus curve of gray as gray candidate pixels; optimizing theset coefficients so that the number of the skin color candidate pixelsand/or gray candidate pixels is maximized and a difference between anaverage color temperature of the group of skin color candidate pixelsand the group of gray candidate pixels is minimized; obtaining a groupof skin color candidate pixels and a group of gray candidate pixels byusing the optimized coefficients; dividing the group of skin colorcandidate pixels into a plurality of subgroups of skin color candidatepixels and dividing the group of gray candidate pixels into a pluralityof subgroups of gray candidate pixels; and estimating a colortemperature of the photographing light source from an average colortemperature of a subgroup of skin color candidate pixels with a highaverage color temperature among the plurality of subgroups of skin colorcandidate pixels and an average color temperature of a subgroup of graycandidate pixels with a high average color temperature among theplurality of subgroups of gray candidate pixels, and wherein the colorimage signals multiplied by the optimized coefficients are corrected bya difference between the estimated color temperature and a colortemperature of reference white.
 8. The white balance correction methodaccording to claim 1, wherein said estimating step includes: multiplyingthe image signals of respective pixels in the input color image by setcoefficients; setting pixels having the multiplied image signals in thevicinity of blackbody locus curve of skin color as skin color candidatepixels and pixels having the multiplied image signals in the vicinity ofa blackbody locus curve of gray as gray candidate pixels; optimizingfirstly the set coefficients so that the number of the skin colorcandidate pixels and the gray candidate pixels is maximized and adifference between an average color temperature of the group of skincolor candidate pixels and an average color temperature of the group ofgray candidate pixels is minimized; estimating a first color temperatureof the photographing light source from the average color temperature ofa group of skin color candidate pixels and the average color temperatureof a group of gray candidate pixels obtained by the first optimization;optimizing secondly the set coefficients so that the number of the graycandidate pixels is maximized and a difference between an average colortemperature of the group of skin color candidate pixels and an averagecolor temperature of the group of gray candidate pixels is minimized;and estimating a second color temperature of the photographing lightsource from the average color temperature of a group of skin colorcandidate pixels and the average color temperature of a group of graycandidate pixels obtained by the second optimization, and wherein thecolor image signals multiplied by the optimized coefficients arecorrected by using both a first white balance correction signal and asecond white balance correction signal, the first white balancecorrection signal being adapted for correcting the color image signalsby a difference between the first estimated color temperature and acolor temperature of reference white, and the second white balancecorrection signal being adapted for correcting them by a differencebetween the second estimated color temperature and a color temperatureof reference white.
 9. The white balance correction method according toclaim 1, wherein said estimating step includes: multiplying the imagesignals of respective pixels in the input color image by setcoefficients; setting pixels having the multiplied image signals in thevicinity of blackbody locus curve of skin color as skin color candidatepixels and pixels having the multiplied image signals in the vicinity ofa blackbody locus curve of gray as gray candidate pixels; optimizingfirstly the set coefficients so that the number of the gray candidatepixels is maximized and a difference between an average colortemperature of the group of skin color candidate pixels and an averagecolor temperature of the group of gray candidate pixels is minimized;estimating a first color temperature of the photographing light sourcefrom the average color temperature of a group of skin color candidatepixels and the average color temperature of a group of gray candidatepixels obtained by the first optimization; optimizing secondly the setcoefficients so that the number of the skin color candidate pixels ismaximized and a difference between an average color temperature of thegroup of skin color candidate pixels and an average color temperature ofthe group of gray candidate pixels is minimized; estimating a secondcolor temperature of the photographing light source from the averagecolor temperature of a group of skin color candidate pixels and theaverage color temperature of a group of gray candidate pixels obtainedby the second optimization, and wherein the color image signalsmultiplied by the optimized coefficients are corrected by using both afirst white balance correction signal and a second white balancecorrection signal, the first white balance correction signal beingadapted for correcting the color image signals by a difference betweenthe first estimated color temperature and a color temperature ofreference white, and the second white balance correction signal beingadapted for correcting them by a difference between the second estimatedcolor temperature and a color temperature of reference white.
 10. Thewhite balance correction method according to claim 1, wherein the imagesignals of respective pixels in the input color image are multiplied byset coefficients and, as a result of the multiplication, when thecoefficients are optimized so that a set objective function isminimized, a maximum value of the image signal of the input image isdetected, and an image signal is used which is standardized so that themaximum value of the image signal becomes 1.0 by dividing each imagesignal of the input image by the maximum value.
 11. The white balancecorrection method according to claim 2, wherein, when each of theblackbody locus curve of skin color and the blackbody locus curve ofgray is set, a spectral sensitivity of a photographing apparatus used toform the input color image is used as a spectral sensitivitydistribution.
 12. The white balance correction method according to claim2, wherein, when each of the blackbody locus curve of skin color and theblackbody locus curve of gray is set, a spectral sensitivity of BT709 isused as a spectral sensitivity distribution.
 13. A white balancecorrection apparatus for correcting white balance when digital imageprocessing is performed on an input color image to form a print,comprising: means for estimating, by using at least gray and/or skincolor information contained in the input color image, a colortemperature of a photographing light source with which the color imagehas been taken; and means for correcting image signals of the colorimage based on the estimated color temperature.
 14. The white balancecorrection apparatus according to claim 13, wherein said means forestimating a color temperature of the photographing light sourceincludes: means for multiplying the image signals of respective pixelsin the input color image by set coefficients; skin color candidate pixeldetection means for detecting pixels having image signals in thevicinity of a blackbody locus curve of skin color as a result of themultiplication and gray candidate pixel detection means for detectingpixels having image signals in the vicinity of a blackbody locus curveof gray as a result of the multiplication; means for optimizing thecoefficients so that the number of the skin color candidate pixelsand/or the number of the gray candidate pixels are maximized and adifference between an average color temperature of the group of skincolor candidate pixels and an average temperature of the group of graycandidate pixels is minimized; and means for computing a colortemperature of the photographing light source from the average colortemperature of the group of skin color candidate pixels and the averagecolor temperature of the group of gray candidate pixels, and wherein themeans for correcting an image signal of the color image is means forcorrecting the color image signals multiplied by the optimizedcoefficients by a difference between the estimated color temperature anda color temperature of reference white.
 15. The white balance correctionapparatus according to claim 13, wherein said means for estimating acolor temperature of the photographing light source includes:coefficient multiplication means for multiplying the image signals ofrespective pixels in the input color image by set coefficients; skincolor candidate pixel detection means for detecting pixels having themultiplied image signals in the vicinity of a blackbody locus curve ofskin color as a result of the multiplication, and gray candidate pixeldetection means for detecting pixels having the multiplied image signalsin the vicinity of a blackbody locus curve of gray as a result of themultiplication; optimization means for optimizing the set coefficientsso that the number of the skin color candidate pixels and the number ofthe gray candidate pixels are maximized and a difference between anaverage color temperature of the skin color candidate pixels and anaverage color temperature of the gray candidate pixels is minimized; andcolor temperature estimating and computing means for estimating thecolor temperature of the photographing light source for a group of skincolor candidate pixels and a group of gray candidate pixels obtained byoptimizing the set coefficients by the optimization means for optimizingthe set coefficients in which the group of skin color candidate pixelsis divided into a plurality of subgroups of skin color candidate pixelsand the group of gray candidate pixels is divided into a plurality ofsubgroups of gray candidate pixels, and a color temperature of thephotographing light source is estimated from an average colortemperature of a subgroup of skin color candidate pixels with a highaverage color temperature among the plurality of subgroups of skin colorcandidate pixels and an average color temperature of a subgroup of graycandidate pixels with a high average color temperature among theplurality of subgroups of gray candidate pixels, and wherein said meansfor correcting the image signals of the color image is means forcorrecting the image signals of the color image multiplied by theoptimized set coefficients by an amount corresponding to a differencebetween the estimated color temperature and a color temperature ofreference white.
 16. A density correction method, comprising the stepsof: multiplying image signals of respective pixels in an input colorimage by set coefficients to detect pixels having the multiplied imagesignals in the vicinity of a blackbody locus curve of skin color as skincolor candidate pixels; and assigning an average obtained forpredetermined color signals from the skin color candidate pixelsdetected to a predetermined density of a color corresponding to thecolor signals on a print.
 17. The density correction method according toclaim 16, wherein the predetermined color signals are G signals and anaverage G signal obtained from the skin color candidate pixels detectedis assigned to a predetermined G density on a print.
 18. The densitycorrection method according to claim 17, wherein said predetermined Gdensity is 0.7 to 1.0.
 19. A recording medium on which one or both of awhite balance correction method and a density correction method arerecorded in a computer-readable manner as a program to be executed by acomputer, wherein said white balance correction method comprises thesteps of: estimating, by using at least gray and/or skin colorinformation contained in an input color image, a color temperature of aphotographing light source with which the color image has been taken;and correcting image signals of the color image based on the estimatedcolor temperature, and wherein said density correction method comprisesthe steps of: multiplying image signals of respective pixels in an inputcolor image by set coefficients to detect pixels having the multipliedimage signals in the vicinity of a blackbody locus curve of skin coloras skin color candidate pixels; and assigning an average obtained forpredetermined color signals from the skin color candidate pixelsdetected to a predetermined density of a color corresponding to thecolor signals on a print.